Advanced Automatic Rig Creation Processes

ABSTRACT

The disclosure provides methods and systems for automatically generating an animatable object, such as a 3D model. In particular, the present technology provides fast, easy, and automatic animatable solutions based on unique facial characteristics of user input. Various embodiments of the present technology include receiving user input, such as a two-dimensional image or three-dimensional scan of a user&#39;s face, and automatically detecting one or more features. The methods and systems may further include deforming a template geometry and a template control structure based on the one or more detected features to automatically generate a custom geometry and custom control structure, respectively. A texture of the received user input may also be transferred to the custom geometry. The animatable object therefore includes the custom geometry, the transferred texture, and the custom control structure, which follow a morphology of the face.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present continuation application claims the priority benefit of U.S.Non-Provisional patent application Ser. No. 17/037,418 filed on Sep. 29,2020 and titled “Additional Developments to the Automatic Rig CreationProcess,” which in turn claims the priority benefit of U.S.Non-Provisional patent application Ser. No. 15/905,667 filed on Feb. 26,2018 and titled “Automatic Rig Creation Process,” the disclosures ofwhich are incorporated by reference in their entireties.

FIELD

The present technology relates generally to animatable 3D models, andmore particularly to systems and methods for automatically generatingcustom meshes and rigging for animatable 3D models.

BACKGROUND

The approaches described in this section could be pursued, but are notnecessarily approaches that have previously been conceived or pursued.Therefore, unless otherwise indicated, it should not be assumed that anyof the approaches described in this section qualify as prior art merelyby virtue of their inclusion in this section.

An animatable 3D model of a virtual character is a computer graphicrepresentation having a geometry or mesh, which may be controlled by arig or control structure. The rig or control structure attaches to areasof the mesh, and affects those areas of the mesh in accordance withgeometric operations applied.

Conventionally, facial animation is done through motion capture and/ormanually by skilled artists, who carefully manipulate animation controlsto create the desired motion of the facial model. Even with the use ofrigs or control structures, the particular process of manipulating therigs to produce realistic and believable facial movements is difficultand dependent upon minute manipulation by animation experts. Since eachface is unique, a mesh and rig of each 3D facial model must beindividually customized for the particular desired facial structure.

Current processes for creating rigs for animation are time consuming,costly, and dependent upon subjective human involvement. As such, a longfelt need exists for automatic and objective animatable solutions tocreate 3D objects including 3D facial models.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described in the Detailed Descriptionbelow. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

According to some embodiments, the present disclosure is directed to amethod for automatically generating an animatable object. The method mayinclude receiving user input; automatically detecting one or morefeatures of the received user input; deforming a template geometry basedon the one or more detected features to automatically generate a customgeometry; transferring a texture of the received user input to thecustom geometry; deforming a template control structure based on the oneor more detected features to automatically generate a custom controlstructure; and generating an animatable object having the customgeometry, the transferred texture, and the custom control structure.

The present technology is also directed to systems for automaticallygenerating an animatable object. In various embodiments, a system mayinclude a processor and a memory for storing executable instructions,the processor executing the instructions to: receive user inputindicative of a face; automatically detect one or more features of thereceived user input; deform a template geometry based on the one or moredetected features to automatically generate a custom geometry; transfera texture of the received user input to the custom geometry; deform atemplate control structure based on the one or more detected features toautomatically generate a custom control structure; and generate ananimatable object having the custom geometry, the transferred texture,and the custom control structure.

According to one or more exemplary embodiments, methods forautomatically generating an animatable 3D model may include receivinguser input; automatically detecting one or more features of the receiveduser input; determining one or more first spatial coordinates, eachfirst spatial coordinate associated with one of the one or more detectedfeatures; deforming a template geometry based on the one or more firstspatial coordinates to automatically generate a custom geometry;transferring a texture of the received user input to the customgeometry; deforming a template control structure based on a subset ofthe one or more spatial coordinates to automatically generate a customcontrol structure; and generating an animatable 3D model having thecustom geometry, the transferred texture, and the custom controlstructure.

Additional objects, advantages, and novel features of the examples willbe set forth in part in the description which follows, and in part willbecome apparent to those skilled in the art upon examination of thefollowing description and the accompanying drawings or may be learned byproduction or operation of the examples. The objects and advantages ofthe concepts may be realized and attained by means of the methodologies,instrumentalities and combinations particularly pointed out in theappended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example, and not by limitation inthe figures of the accompanying drawings, in which like referencesindicate similar elements.

FIG. 1 is a schematic diagram of an example system architecture forpracticing aspects of the present disclosure.

FIG. 2 is a block diagram for automatically generating an animatableobject, according to embodiments of the present disclosure.

FIG. 3 is a flowchart of an example method for automatically generatingan animatable object, according to embodiments of the presentdisclosure.

FIG. 4 is an example user input having one or more facial featuresdetected via autolandmarking.

FIG. 5 is a rendering of an exemplary animatable 3D model created fromthe example user input of FIG. 4 .

FIG. 6 is a real-time rendering of the animatable 3D model of FIG. 5 ina virtual gaming environment.

FIG. 7 is a schematic diagram of an example computer device that can beutilized to implement aspects of the present disclosure.

FIG. 8 shows a light non-uniformity detection example.

FIG. 9 shows an example of shadow correction and specular correction.

FIG. 10 shows a normal map and an ambient occlusion map for a specificinput image.

FIG. 11 shows an exemplary red-green-blue image and its depth channelcomponents.

FIG. 12 shows a comparison of facial reconstruction methods.

FIG. 13 shows an example of a full point cloud of a human head computedfrom a mobile scanner.

FIG. 14 shows an example of textured mesh computed from a mobilescanner.

DETAILED DESCRIPTION

The following detailed description includes references to theaccompanying drawings, which form a part of the detailed description.The drawings show illustrations in accordance with example embodiments.These example embodiments, which are also referred to herein as“examples,” are described in enough detail to enable those skilled inthe art to practice the present subject matter. The embodiments can becombined, other embodiments can be utilized, or structural, logical, andelectrical changes can be made without departing from the scope of whatis claimed. The following detailed description is therefore not to betaken in a limiting sense, and the scope is defined by the appendedclaims and their equivalents.

In general, various embodiments of the present disclosure are directedto fast, easy, and automatic animatable solutions for generatingthree-dimensional (3D) objects. For example, one or more embodimentsinclude automatically generating a 3D model from a user input, such as atwo-dimensional (2D) photograph or 3D scan data of a user's face. The 3Dmodel may include a custom geometry (e.g. mesh), a texture, and a customcontrol structure (e.g. rig) based on the user input. These and otheradvantages of the present disclosure are provided herein in greaterdetail with reference to the drawings.

FIG. 1 illustrates an exemplary architecture 100 for practicing aspectsof the present disclosure. The architecture comprises one or moreclients 105 communicatively coupled to a server system 110 via a publicor private network, such as network 115. In various embodiments, theclient 105 includes at least one of a personal computer, a laptop, aSmartphone, or other suitable computing device.

Suitable networks for network 115 may include or interface with any oneor more of, for instance, a local intranet, a PAN (Personal AreaNetwork), a LAN (Local Area Network), a WAN (Wide Area Network), a MAN(Metropolitan Area Network), a virtual private network (VPN), a storagearea network (SAN), a frame relay connection, an Advanced IntelligentNetwork (AIN) connection, a synchronous optical network (SONET)connection, a digital T1, T3, E1 or E3 line, Digital Data Service (DDS)connection, DSL (Digital Subscriber Line) connection, an Ethernetconnection, an ISDN (Integrated Services Digital Network) line, adial-up port such as a V.90, V.34 or V.34bis analog modem connection, acable modem, an ATM (Asynchronous Transfer Mode) connection, or an FDDI(Fiber Distributed Data Interface) or CDDI (Copper Distributed DataInterface) connection. Furthermore, communications may also includelinks to any of a variety of wireless networks, including WAP (WirelessApplication Protocol), GPRS (General Packet Radio Service), GSM (GlobalSystem for Mobile Communication), CDMA (Code Division Multiple Access)or TDMA (Time Division Multiple Access), cellular phone networks, GPS(Global Positioning System), CDPD (cellular digital packet data), RIM(Research in Motion, Limited) duplex paging network, Bluetooth radio, oran IEEE 802.11-based radio frequency network. The network 115 canfurther include or interface with any one or more of an RS-232 serialconnection, an IEEE-1394 (Firewire) connection, a Fiber Channelconnection, an IrDA (infrared) port, a SCSI (Small Computer SystemsInterface) connection, a USB (Universal Serial Bus) connection or otherwired or wireless, digital or analog interface or connection, mesh orDigi® networking.

Generally, the server system 110 is configured to provide variousfunctionalities which are described in greater detail throughout thepresent disclosure. In various embodiments, the server system 110comprises a processor 120, a memory 125, and network interface 130.According to some embodiments, the memory 125 comprises logic 135(otherwise referred to as instructions) that may be executed by theprocessor 120 to perform various methods described herein. For example,the logic 135 may include autolandmarking module 140, retopology module145, texture transfer module 150, and rigging module 155, which areconfigured to provide some or all of the functionalities described ingreater detail herein. It is to be understood that, while the methodsdescribed herein are generally attributed to the server system 110, mayalso be executed by the client 105. In other embodiments, the serversystem 110 and client 105 may cooperate to provide the functionalitiesdescribed herein. The client 105 may be provided with a client-sideapplication that interacts with the server system 110 in a client/serverrelationship.

In general, the autolandmarking module 140 may receive user input, forexample in the form of a 2D image or 3D data associated with a face orhead of a person, from the client 105. The autolandmarking module 140may automatically detect facial features (e.g. landmarks) from the userinput, which are unique to the face associated with the user input. Invarious embodiments, the automatic detection of facial features is basedon machine learning algorithms on an associated database. In someembodiments, the autolandmarking module 140 casts 2D coordinates of thedetected facial features from a 2D input into 3D coordinates (e.g.spatial coordinates) associated with a template geometry.

In one or more embodiments, the retopology module 145 automaticallydeforms, based on the detected facial features, the template geometry tocreate a custom geometry. The template geometry may have a pre-definedset of facial features with associated coordinates. In general, verticesof the custom geometry follow a morphology of the original faceassociated with the user input.

In some embodiments, the texture transfer module 150 determines atexture from the user input. In general, the texture transfer module 150uses the user input as the texture, such as the 2D image or surfaceinformation of 3D input data. The texture transfer module 150 may matchthe texture to the custom geometry created by the retopology module. Ingeneral, the texture is not modified—for example, no resampling isperformed and no average is performed of the deformed space in anydatabase. Advantageously, the custom geometry has already been deformedto match the texture given in the user input. The texture transfermodule 150 may automatically map the texture to the custom geometry byassociating each pixel in the user input to a corresponding vertex orsurface on the custom geometry. Moreover, the transferred texture isconfigured to adapt to an environment of the 3D model, for example, toadjust for illumination and context.

In various embodiments, the rigging module 155 automatically generates acustom control structure based on the detected facial features. Thecontrol structure generally provides elements that allow manipulation ofthe custom geometry to create animation. The control structure may adaptand create key elements to deform the custom geometry during animationsuch that behaviors and movements are smooth and follow the morphologyof the face associated with the user input. In particular, in someembodiments, the rigging module 155 deforms a template control structurebased on control elements determined from the detected facial features.In general, the texture transfer module 150 and the rigging module 155may operate in series or in parallel.

The server system 110 may then combine the custom geometry, thetransferred texture, and the custom control structure to automaticallygenerate the animatable 3D model. Thus, the final output may includeaccurate landmark detection, an accurate custom geometry that follows amorphology of the face associated with the user input, accurate texturemapping, and a custom control structure that allows smooth and accuratesimulation of behavior and movement of the 3D model.

Another aspect of the present disclosure is that the server system 110may utilize any template, or pre-defined, geometry and any template, orpre-defined, control structure. For example, the user may input, via theclient 105, a user-defined geometry, which includes a pre-defined set offacial features with associated coordinates, to replace the templategeometry in the server system 110. Likewise, the user may input, via theclient 105, a user-defined control structure to replace the templatecontrol structure used by the rigging module 155 to generate the customcontrol structure.

FIG. 2 depicts a block diagram of a system 200 for automaticallygenerating a custom animatable object, according to embodiments of thepresent disclosure.

At block 210, user input is received that is indicative of a face of theuser. The user input may be an image, a frame of a video, a 3D scan, orother suitable media. It is to be understood that the user input mayfurther comprise body information of the user. In such embodiments, thetemplate geometry and template control structure would compriseapproximations of the appearance and control elements of a humanoidbody, and the one or more features detected by autolandmarking wouldinclude further features indicative of the body.

At block 220, autolandmarking is performed to automatically detectfacial features from the user input from block 210. The facial featuresare unique to the user. If the user input is an image or in another 2Dformat, the facial features are detected and stored as 2D coordinates,for example those shown and described in FIG. 4 . The 2D coordinates areconverted into spatial coordinates using ray casting techniques, orother suitable algorithms. For example, the system 200 may includecreating an artificial 2D plane, having the user input and detectedfeatures, in front of a template 3D model. It is to be understood thatthe template 3D model may include a template, or generic, geometry andtemplate, or generic, control structure. An origin coordinate isdetermined based on a spatial position of the user input and thetemplate 3D model. Using ray casting techniques, each detected featureis projected from the artificial 2D plane onto the template 3D model viaa ray passing from the origin through the respective 2D coordinate ofthe detected feature. The projection results in a spatial coordinateindicative of where the detected feature should be for the customanimatable model. The depth of each spatial coordinate, as well as therelative position of the artificial 2D plane, template 3D model, andorigin coordinate, may be automatically determined based on predictionsand statistics of facial morphology. In other embodiments, the depth ofeach spatial coordinate is pre-defined in the template geometry.

In various embodiments, the server comprises a template 3D model havinga template geometry and a template control structure, also referred toas a pre-defined geometry and a pre-defined control structure,respectively. The template geometry is an approximation of what theresulting facial mesh should look like, although it is to be understoodthat the template geometry may be any suitable size or shape. Thetemplate control structure may be any suitable rig for controllingmovement of a geometry, such as a bone-based rig, blend-shape rig,free-form deformer, physically-based model, or other suitable controlstructure. For example, the template control structure may comprise apre-defined set of bones that will create facial movements that followthe morphology and behavior a face of the template geometry.

At block 230, retopology is performed to deform the template geometrybased on the detected facial features. The template geometry may includea set of template facial features that correspond to facial featuresdetected in the autolandmarking in block 220. As such, spatialcoordinates of the detected facial features are matched to correspondingspatial coordinates of the template facial features. Based on thematching, the template geometry is automatically deformed to create acustom geometry using radial basis functions, or other suitablealgorithms. Advantageously, the custom geometry is clean. That is,vertices of the custom geometry follow a morphology of the face from theuser input.

In some embodiments, block 230 includes dynamically deforming thetemplate geometry based on a determined type of facial structureassociated with the user input. For example, the server may comprise aplurality of template geometries, each template geometry correspondingto a different type of facial structure. The different types of facialstructures may relate to different genders or races, for example, andreflect differences in statistical information regarding the facialmorphologies of each group. As such, each template geometry may comprisedifferent spatial coordinates for the set of generic facial features.Block 230 may further include determining which template geometry of theplurality of template geometries most closely matches the one or moredetected features of the received user input, and using that templategeometry.

Likewise, the server may comprise a plurality of template models, eachwith different parameters for different target applications. Forexample, a first template model of the template models may be configuredwith a cinematic rig with a large number of control points and highpolygon count, while a second template model of the template models maybe configured for a lightweight, non-playable character in a video gamewith a few control points and a low polygon count. The server may selectwhich template model to use based on user input or automatically.

Block 240 and block 250 may be performed in series or in parallel, asshown in FIG. 2 . At block 240, a texture of the user input istransferred to the custom geometry automatically generated in block 230.Transferring the texture may include mapping a plurality of pixels ofthe user input to vertices of the custom geometry. At block 250, riggingis performed to automatically generate a custom control structure basedon the detected facial features and the template control structure. Thetemplate control structure may include a pre-defined set of controlelements, such as bones in a bone-based rig, associated with spatialcoordinates. A subset of the detected facial features may be associatedwith control elements, herein after referred to as detected controlelements of the user input. As such, spatial coordinates of the detectedcontrol elements are matched to corresponding spatial coordinates of thetemplate control structure. Based on the matching, the template controlstructure is automatically deformed to create a custom control structureusing radial basis functions, or other suitable algorithms.Advantageously, one or more algorithms used to deform the templatecontrol structure may be the same as the one or more algorithms used todeform the template geometry. The custom control structure provides theelements to allow for the manipulation and animation of the customgeometry, and is configured to follow the morphology of the face fromthe user input.

At block 260, an animatable output is automatically generated from thecustom geometry, the transferred texture, and the custom controlstructure from blocks 230, 240, and 250. Thus the animatable objectcomprises a deformable, custom geometry that uses a custom controlstructure to generate behaviors and movement. The custom geometry, thetransferred texture, and the custom control structure are all based onthe user input, and thus are personalized to the unique face of the userindicative of the user input. For example, the animatable object may bea 3D model of a humanoid head having the face and morphology of theuser. It is to be understood that the same methods may be applied toother physical structures, such as a body of the user. In suchembodiments, the template geometry and template control structure wouldcomprise approximations of the appearance and control elements of ahumanoid body, and the feature detected by autolandmarking would includefurther features indicative of the body.

Advantageously, embodiments of the present disclosure are versatile andallow the user to input a user-defined template geometry and/or auser-defined template control structure, which are then used in theautomatic system. If the user wants a mesh with fewer polygons or wouldlike a control structure set up for motion capture instead of keyframeanimation, for example, the user may input such a template geometry ortemplate control structure into the system.

At optional block 270, a user-defined geometry is received. The servermay store the user-defined geometry and associate the user-definedgeometry with the user for future use. At block 230, the system maydetermine whether a user-defined geometry is stored for the user. Basedon the determination, the user-defined geometry is deformed instead ofthe template geometry using the same methodology. In variousembodiments, the system determines whether the user-defined geometrycomprises the same features as the template geometry. Based on thedetermination, the system may dynamically and automatically adjust thefeatures detected during autolandmarking in block 220, such that thedetected features correspond to the features present in the user-definedgeometry.

At optional block 280, a user-defined control structure is received. Theuser-defined control structure may be configured to control the behaviorand movement of the user-defined geometry. The server may store theuser-defined control structure and associate the user-defined controlstructure with the user for future use. At block 250, the system maydetermine whether a user-defined control structure is stored for theuser. Based on the determination, rigging is performed to deform theuser-defined control structure instead of the template control structureusing the same methodology.

In one or more embodiments, the animatable object is dynamically andautomatically generated in real-time based on a dynamic user input, forexample from a video signal from a camera system. In such embodiments,the system would perform the autolandmarking, retopology, texturetransfer, and rigging steps in real-time to dynamically andautomatically generate the custom geometry, transferred texture, andcustom control structure. For example, the system may capture featuresof the user via autolandmarking of the dynamic user input, and map thefeatures to both the custom geometry and the custom control structure tocreate the animated 3D model. Control elements of the custom controlstructure are configured to allow the 3D model to move according to themorphology of the user. Real-time mapping of the features to the controlstructure allow for smooth manipulation of the custom geometry inreal-time.

FIG. 3 is a flow chart showing an exemplary method 300 for automaticgeneration of an animatable object. Method 300 can be performed byprocessing logic that includes hardware (e.g. decision-making logic,dedicated logic, programmable logic, application-specific integratedcircuit), software (such as software run on a general-purpose computersystem or dedicated machine), or a combination of both. In one exampleembodiment, the processing logic refers to one or more elements thesystems shown in FIGS. 1-2 .

Operations of method 300 recited below can be implemented in an orderdifferent than described and shown in FIG. 3 . Moreover, the method 300may have additional operations not shown herein, but which can beevident to those skilled in the art from the present disclosure. Method300 may also have fewer operations than shown in FIG. 3 and describedbelow.

The method 300 may commence in operation 310 with receiving user inputindicative of a face. In various embodiments, the user input includes atleast one of an image, a video signal, and a 3D scan, which may beindicative of a face and/or body of a user. In certain embodiments, theuser input is received from a client device via a network. It is to beunderstood that each operation of the method 300 may be performed inreal-time, such that a dynamic user input such as a video signal ispermitted to be input to automatically generate a dynamic 3D model thatfollows a morphology of the user input in real-time.

Operation 320 includes automatically detecting one or more features ofthe received user input. The automatically detecting the one or morefeatures may include determining a set of spatial coordinates via raycasting techniques, each spatial coordinate associated with one of theone or more features of the received user input. In one or moreembodiments, operation 320 includes casting a two-dimensional coordinateof each of the one or more detected features onto a template geometryusing the ray casting techniques.

The method 300 may proceed in operation 330 with deforming a templategeometry based on the one or more detected features to automaticallygenerate a custom geometry. In some embodiments, a set of features ofthe template geometry corresponds to the one or more detected features.The deforming the template geometry may include matching the spatialcoordinates of the one or more detected features to the set of featuresof the template geometry, and based on the matching, applying a radialbasis function to the spatial coordinates of the one or more detectedfeatures and the set of features of the template geometry. Theapplication of the radial basis function may produce vertices of thecustom geometry which are based on the spatial coordinates of the one ormore detected facial features.

In certain embodiments, the template geometry is a user-defined geometryreceived from a client device. The method 300 may further includestoring the user-defined geometry as being associated with the clientdevice.

Operation 340 and operation 350 may be performed in parallel, as shownin FIG. 3 . Operation 340 may include transferring a texture of thereceived user input to the custom geometry. In certain embodiments, thetransferring the texture to the custom geometry includes automaticallymapping at least one pixel of the texture to a corresponding vertex onthe custom geometry.

In various embodiments, operation 350 includes deforming a templatecontrol structure based on the one or more detected features toautomatically generate a custom control structure. A set of controlelements of the template control structure may correspond to a subset ofthe one or more detected features. The deforming the template controlstructure may include matching the subset of the one or more detectedfeatures to the set of control elements of the template controlstructure, and based on the matching, applying a radial basis functionto the subset of the one or more detected features and the set ofcontrol elements. The application of the radial basis function mayproduce control elements of the custom control structure which are basedon spatial coordinates of the subset of the one or more detected facialfeatures.

In certain embodiments, the template control structure is a user-definedcontrol structure received from a client device. The method 300 mayfurther include storing the user-defined control structure as beingassociated with the client device.

At operation 360, an animatable object is automatically generated havingthe custom geometry, the transferred texture, and the custom controlstructure.

FIG. 4 is screenshot of an example user input 400 indicative of a user'sface 410 having one or more facial features 420, 430, 440 detected viaautolandmarking. Each of the one or more detected facial features 420,430, 440 is represented by a circle over the user input 400, though forease of illustration only some of the one or more detected facialfeatures 420, 430, 440 are marked with a reference number. The one ormore detected facial features 420, 430, 440 may be described as a set ofrules which control the automatic generation of the custom geometry andcustom control structure and configure the resulting animatable 3D modelto follow the morphology of the face 410. In one or more embodiments, afirst set of facial features 420 may be used in the deformation of thetemplate geometry to the custom geometry. A second set of facialfeatures 430 may facilitate alignment and scale, while a third set offacial features 440 may be used to determine coloring (e.g. eyecoloring). In such an example, the set of facial features for only oneeye may be necessary to determine the eye color. It is to be understoodthat the identification of any particular detected facial feature 420,430, 440 in FIG. 4 is exemplary and different combinations of detectedfacial features, and designation of the type of detected facialfeatures, are contemplated by the present disclosure.

FIGS. 5 and 6 illustrate exemplary animatable objects created from themethods described in the present disclosure. FIG. 5 is a rendering of anexemplary animatable 3D model 500 created from the example user input400 of FIG. 4 . Moreover, FIG. 6 depicts a real-time rendering of ananimatable 3D model 500 of FIG. 5 in a virtual gaming environment 600.

FIG. 7 illustrates an exemplary computer system 700 that may be used toimplement some embodiments of the present technology. Computer system700 may be implemented in the contexts of the likes of computing systemssuch as server system 110 and client 107. Computer system 700 includesone or more processor units 710 and main memory 720. Main memory 720stores, in part, instructions and data for execution by processor units710. Main memory 720 stores the executable code when in operation, inthis example. Computer system 700 may further include one or more of amass data storage 730, portable storage device 740, output devices 750,user input devices 760, a graphics display system 770, and peripheraldevices 780.

The components shown in FIG. 7 are depicted as being connected via asingle bus 790. The components may be connected through one or more datatransport means. Processor unit 710 and main memory 720 is connected viaa local microprocessor bus, and the mass data storage 730, peripheraldevice(s) 780, portable storage device 740, and graphics display system770 are connected via one or more input/output (I/O) buses.

Mass data storage 730, which can be implemented with a magnetic diskdrive, solid state drive, or an optical disk drive, is a non-volatilestorage device for storing data and instructions for use by processorunit 710. Mass data storage 730 stores the system software forimplementing embodiments of the present disclosure for purposes ofloading that software into main memory 720.

Portable storage device 740 operates in conjunction with a portablenon-volatile storage medium, such as a flash drive, floppy disk, compactdisk, digital video disc, or USB storage device, to input and outputdata and code to and from computer system 700. The system software forimplementing embodiments of the present disclosure is stored on such aportable medium and input to computer system 700 via portable storagedevice 740.

User input devices 760 can provide a portion of a user interface. Userinput devices 760 may include one or more microphones, an alphanumerickeypad, such as a keyboard, for inputting alphanumeric and otherinformation, or a pointing device, such as a mouse, a trackball, stylus,or cursor direction keys. User input devices 760 can also include atouchscreen. Additionally, computer system 700 includes output devices750. Suitable output devices 750 include speakers, printers, networkinterfaces, and monitors.

Graphics display system 770 include a liquid crystal display (LCD) orother suitable display device. Graphics display system 770 isconfigurable to receive textual and graphical information and processesthe information for output to the display device. Peripheral devices 780may include any type of computer support device to add additionalfunctionality to the computer system.

The components provided in computer system 700 are those typically foundin computer systems that may be suitable for use with embodiments of thepresent disclosure and are intended to represent a broad category ofsuch computer components that are well known in the art. Thus, computersystem 700 can be a personal computer (PC), hand held computer system,telephone, mobile computer system, workstation, tablet computer, mobilephone, server, minicomputer, mainframe computer, wearable computer, orany other computing system. The computer may also include different busconfigurations, networked platforms, multi-processor platforms, and thelike.

Some of the above-described functions may be composed of instructionsthat are stored on storage media (e.g., computer-readable medium). Theinstructions may be retrieved and executed by the processor. Someexamples of storage media are memory devices, tapes, disks, and thelike. The instructions are operational when executed by the processor todirect the processor to operate in accord with the technology. Thoseskilled in the art are familiar with instructions, processor(s), andstorage media.

In some embodiments, computer system 700 may be implemented as acloud-based computing environment, such as a virtual machine operatingwithin a computing cloud. In other embodiments, computer system 700 mayitself include a cloud-based computing environment, where thefunctionalities of the computer system 700 are executed in a distributedfashion. Thus, computer system 700, when configured as a computingcloud, may include pluralities of computing devices in various forms, aswill be described in greater detail below.

In general, a cloud-based computing environment is a resource thattypically combines the computational power of a large grouping ofprocessors (such as within web servers) and/or that combines the storagecapacity of a large grouping of computer memories or storage devices.Systems that provide cloud-based resources may be utilized exclusivelyby their owners or such systems may be accessible to outside users whodeploy applications within the computing infrastructure to obtain thebenefit of large computational or storage resources.

The cloud is formed, for example, by a network of web servers thatcomprise a plurality of computing devices, such as computer system 700,with each server (or at least a plurality thereof) providing processorand/or storage resources. These servers manage workloads provided bymultiple users (e.g., cloud resource customers or other users).Typically, each user places workload demands upon the cloud that vary inreal-time, sometimes dramatically. The nature and extent of thesevariations typically depends on the type of business associated with theuser.

It is noteworthy that any hardware platform suitable for performing theprocessing described herein is suitable for use with the technology. Theterms “computer-readable storage medium” and “computer-readable storagemedia” as used herein refer to any medium or media that participate inproviding instructions to a CPU for execution. Such media can take manyforms, including, but not limited to, non-volatile media, volatile mediaand transmission media. Non-volatile media include, for example, opticalor magnetic disks, such as a fixed disk. Volatile media include dynamicmemory, such as system RAM. Transmission media include coaxial cables,copper wire and fiber optics, among others, including the wires thatcomprise one embodiment of a bus. Transmission media can also take theform of acoustic or light waves, such as those generated during radiofrequency (RF) and infrared (IR) data communications. Common forms ofcomputer-readable media include, for example, a floppy disk, a flexibledisk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROMdisk, digital video disk (DVD), any other optical medium, any otherphysical medium with patterns of marks or holes, a RAM, a PROM, anEPROM, an EEPROM, a FLASHEPROM, any other memory chip or data exchangeadapter, a carrier wave, or any other medium from which a computer canread.

Various forms of computer-readable media may be involved in carrying oneor more sequences of one or more instructions to a CPU for execution. Abus carries the data to system RAM, from which a CPU retrieves andexecutes the instructions. The instructions received by system RAM canoptionally be stored on a fixed disk either before or after execution bya CPU.

FIG. 8 shows a light non-uniformity detection example.

According to various exemplary embodiments, a three-dimensional (“3D”)facial reconstruction pipeline comprises a light correction componentcomposed of a set of automated methods that perform detection andcorrection of light artifacts on an input image. The input image isusually a facial texture in the UV space and the component operates uponthe skin pixels in the image. As such, the component also operates onskin and hair masks of the input facial texture in the UV space. Theletters “U” and “V” denote the axes of the 2D texture because “X”, “Y”,and “Z” are already used to denote the axes of the 3D object in modelspace. The exemplary methods also work if the images are not in UVspace.

In the detection stage, the exemplary methods perform detection ofshadows and specular artifacts independently and using traditionalcomputer vision techniques based on statistical analysis. FIG. 8 showsthe specular mask 805 and shadow mask 810 produced in the detectionstage for the region of interest (“ROI”) of a specific input image 800.

FIG. 9 shows an example of shadow correction and specular correction.

In the correction stage, exemplary methods perform brightness correctionand color adjustment based on the average color of the skin pixels whichare not affected by light artifacts. The two corrections are appliedsequentially. The hair mask is used in this phase to ensure that hairpixels do not interfere in the selection of “clean” skin pixels. Thisstage also relies on traditional statistic-based computer visiontechniques. FIG. 9 shows the ROI of an inputted face image 900, theimage 905 after shadow correction, and the image 910 after the specularreflection correction.

FIG. 10 shows a normal map and an ambient occlusion map for a specificinput image.

According to various exemplary embodiments, a three-dimensional (“3D”)facial reconstruction pipeline comprises a shading component thatsynthesizes a normal map and an ambient occlusion map from a given inputtexture in the UV space. The input texture is usually a facial texturein the UV space, but the component would also work if the image is notin UV space. Both maps are produced using computer vision techniques.FIG. 10 shows a face albedo 1000 in UV space, a normal map of the facealbedo 1005 in the UV space, and an ambient occlusion map of the facealbedo 1010 in the UV space.

FIG. 11 shows an exemplary red-green-blue image and its depth channelcomponents.

A three-dimensional (“3D”) facial reconstruction pipeline, according tomost exemplary embodiments, may take a single photo as an input. The 3Dlandmark coordinates used to deform the template face model are theninferred from the 2D pixel coordinates computed in the autolandmarkingstep so as to minimize some measure of geometric discrepancy in 3D withrespect to the template. This process tends to make the reconstructedface similar to the template when looking sideways, as it merelyextrapolates the subject's depth information from the template data.

For higher accuracy, the pipeline was extended to the case where a depthimage acquired jointly with the input photo is available so that theinput is an “RGBD image”, that is, an image featuring the customarythree color channels (red, green, blue) available from a photo, such asphoto 1100, as well as an additional “depth channel,” such as seen indepth image 1105. The depth channel assigns each pixel to the distanceof its corresponding 3D point to the camera principal plane, or to thecamera center of projection. Owing to the extra depth input, the 3Dlandmark coordinates are used to drive facial reconstruction and can becomputed more accurately than using a single photo.

FIG. 12 shows a comparison of facial reconstruction methods.

A two-dimensional (“2D”) facial reconstruction pipeline, according tomost exemplary embodiments, reconstructs a head geometry based onapproximately 140 landmarks extracted automatically from an input photousing a machine vision algorithm. Due to the limited accuracy of depthmeasurements (which is also highly non-uniform in space), the depthvalue cannot be directly read from the depth map for each landmark.Instead, the measured depth data is used as an additional set ofconstraints to the template matching step, assuming measurement errors.

Turning again to FIG. 12 , the comparison of facial reconstructionmethods shows RGB input 1200 and RGBD input 1205. In the experimentsshown in FIG. 12 with data provided by Huawei, a standard error of 0.3cm was assumed on the depth measurement within a certain image region(typically excluding the outer part of the face).

FIG. 13 shows an example of a full point cloud of a human head computedfrom a mobile scanner.

Exemplary embodiments employ real-time acquisition of RGBD data usingboth a front camera and a depth sensor all around a human subject. Someimplementations include the use of a smartphone or a RGBD camera. Thisparticular scanning procedure produces multiple RGBD imagescorresponding to different views of the head, covering up to 360°including the top of the head and chin.

The acquired scans are then submitted to a series of processing stepsincluding converting the scans in real time into a point cloud using aclassical point registration technique, namely the Iterative ClosestPoint algorithm. In addition to this geometric reconstruction, a sampleof pixels is selected from the scan, at specific angles, to produce thefacial texture later used in a 3D facial reconstruction pipeline. FIG.13 shows a full head point cloud acquired using an app in several views:front 1300, right side 1305, back 1310 and left side 1315.

FIG. 14 shows an example of textured mesh computed from a mobilescanner.

Following the conversion of scans into a point cloud, the point cloud isfurther converted into a mesh using an automated method such as Poissonsurface reconstruction. Further mesh processing is applied (removal ofduplicated vertices and triangles, removal of degenerate triangles,removal of non-manifold edges, and removal of unreferenced vertices) toproduce a clean mesh.

Next, a texture material is produced for the mesh using the previouslyselected pixel scan sample. FIG. 14 shows the mesh produced from thehead point cloud for the same scan as before, in several views: front1400, right side 1405, back 1410 and left side 1415. Finally, thetextured mesh computed from the input scans as described above may beinput to the rigging method as an alternative to a single photo, asingle RGBD image, or other types of imaging data. Higher facereconstruction fidelity is expected since the textured mesh encodesdense three-dimensional information without occlusions. The headreconstruction problem then boils down to a pure retopology problem(i.e., deforming the template mesh to match the input mesh).

In other exemplary embodiments, descriptor-based mesh deformation may beperformed. A reconstructed head is produced by deforming an existingtemplate mesh in 3D. In the default method instance, this deformationprocess is accomplished by matching a set of key points from thetemplate head with their equivalent anatomical points detected on theinput subject's head by the autolandmarking component. An alternativeway to drive the template deformation process is to use descriptor-basedtarget points as opposed to image-based points. Specifically, in thisuse case, the target points corresponding to the template key points aredefined by a two-step process. First, the user specifies a set ofdescriptors; these can be quantitative (facial proportions, lengths,age, etc.) or qualitative (“feminine”, “long nose”, “strong jawline”,“slanted eyes”, etc.). Next, the user-supplied descriptors are convertedinto actual 3D point coordinates by automatically selecting head partsthat best match the descriptors in an existing database of head meshes.This step is carried out by the numerical optimization of similaritymeasures corresponding to the relevant head part comparisons. The 3Dcoordinates defined in this manner are then used similarly to thestandard case of image-based landmarks to drive the templatedeformation, resulting in a head mesh that optimally matches the inputdescription. In addition to the possibility to incorporate user-supplieddescriptors, the rig creation process can also output facial descriptorscorresponding to the reconstructed head mesh properties. These areevaluated automatically at the end of the process using point distancecomputation on the key points associated with the final head mesh,and/or using the same type of database matching technique as describedabove to produce qualitative descriptors.

Rigging, retargeting and animation may be performed in further exemplaryembodiments. Facial animation is usually done through automated motioncapture or manually by a skilled artist. An alternative method is toanimate the reconstructed head mesh via artificial intelligence. In thisinstance, the three-dimensional trajectories of the mesh points used todrive the mesh deformation over time are generated by a machine learningalgorithm trained from a set of real-world videos and/or motion capturedata to simulate realistic facial deformations. Such simulated yetplausible deformations can be generated depending on user-specifiedexpression types (e.g. smiling, happy, sad, angry, etc.). This fullyautomated artificial intelligence based technique alleviates the needfor subject-specific temporal data acquisition or manual editing forfacial animation.

In some exemplary embodiments, mobile phone motion capture may beperformed. Text to speech technology can be used to automaticallyconvert a written text input by a user into a sequence of visemes. Suchvisemes are associated with pre-defined facial deformations encoded bytemplate mesh points coordinates. Therefore, the reconstructed head meshmay be animated in synchrony with a synthetically produced audio speech,so that the lips deform while the speech is being produced and thecharacter thus appears to be speaking.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. The descriptions are not intended to limit the scope of thetechnology to the particular forms set forth herein. Thus, the breadthand scope of a preferred embodiment should not be limited by any of theabove-described exemplary embodiments. It should be understood that theabove description is illustrative and not restrictive. To the contrary,the present descriptions are intended to cover such alternatives,modifications, and equivalents as may be included within the spirit andscope of the technology as defined by the appended claims and otherwiseappreciated by one of ordinary skill in the art. The scope of thetechnology should, therefore, be determined not with reference to theabove description, but instead should be determined with reference tothe appended claims along with their full scope of equivalents.

What is claimed is:
 1. A method for generating a full point cloud of a human head, the method comprising: acquiring in real-time red-green-blue-depth scans around a human subject; converting the red-green-blue-depth scans into a point cloud; converting the point cloud into a mesh; selecting a sample of pixels from the scans; using the sample of pixels to produce a facial texture for the mesh; and inputting the textured mesh to a rigging method.
 2. The method of claim 1, further comprising acquiring RGBD data using a front camera.
 3. The method of claim 1, further comprising acquiring RGBD data using a depth sensor.
 4. The method of claim 1, further comprising selecting the sample of pixels from the scans at specific angles.
 5. A method for deforming a template by user-specified descriptors, the method comprising: specifying a set of descriptors; converting the descriptors into three-dimensional coordinates; using the three dimensional coordinates to deform the template to generate a head mesh that matches the set of descriptors.
 6. The method of claim 5, wherein the set of descriptors are quantitative.
 7. The method of claim 5, wherein the set of descriptors are qualitative.
 8. The method of claim 5, wherein the converting the descriptors into the three-dimensional coordinates is by automatically selecting head parts that best match the descriptors in a database of head meshes.
 9. The method of claim 8, the head parts including a facial structure.
 10. The method of claim 9, the facial structure relating to a gender.
 11. The method of claim 9, the facial structure reflecting to a difference in statistical information regarding a facial morphology of a group.
 12. A method for animating a reconstructed head mesh, the method comprising: using artificial intelligence to generate three-dimensional trajectories of mesh points to simulate a facial deformation.
 13. The method of claim 12, further comprising using a user-specified expression type.
 14. The method of claim 13, wherein the user-specified expression type is smiling, happy, sad or angry.
 15. A method for animating a reconstructed head mesh, the method comprising: associating a written text sequence with a viseme; associating the viseme with a predefined facial deformation encoded by a template mesh point coordinate; and animating in synchrony the predefined facial deformation on the reconstructed head mesh with a synthetically produced audio speech.
 16. The method of claim 15, further comprising the reconstructed head mesh having lips deforming while the audio speech is being produced.
 17. The method of claim 16, further comprising the reconstructed head mesh appearing to be speaking.
 18. The method of claim 15, further comprising using automated motion capture.
 19. The method of claim 18, further comprising the automated motion capture resulting in facial animation. 